Unfortunately, there was no way to get the fascia back on and bolted on properly with the driver side duct on. No matter how much I trimmed it wouldn't fit. Because of the size of the intercooler, and the fact that the duct wouldn't set flush up against the face of the core, the outside part where the 2 bolts go that screw into the end tank was keeping the fascia from lining up and bolting on.
I scrapped both ducts as I'm getting the VIS Racing fascia put on in about a week anyway and will buy the JDM ducts for that and give it another shot then.